SUMMARY: This project aims to construct a predictive model using a TensorFlow convolutional neural network (CNN) and document the end-to-end steps using a template. The American Sign Language Alphabet Dataset is a multi-class classification situation where we attempt to predict one of several (more than two) possible outcomes.
INTRODUCTION: The dataset contains over 22,000 images of alphabets from American Sign Language, separated into 29 folders that represent the various classes. The research team collected these images to investigate the possibilities of reducing the communication gap between sign-language users and non-Sign language users.
ANALYSIS: The Xception model's performance achieved an accuracy score of 99.23% after three epochs using the training dataset. When we applied the model to the validation dataset, the model achieved an accuracy score of 89.94%.
CONCLUSION: In this iteration, the TensorFlow Xception CNN model appeared suitable for modeling this dataset.
Dataset ML Model: Multi-Class classification with numerical features
Dataset Used: American Sign Language Alphabet Dataset
Dataset Reference: https://www.kaggle.com/datasets/debashishsau/aslamerican-sign-language-aplhabet-dataset
One source of potential performance benchmarks: https://www.kaggle.com/datasets/debashishsau/aslamerican-sign-language-aplhabet-dataset/code
# Retrieve CPU information from the system
ncpu = !nproc
print("The number of available CPUs is:", ncpu[0])
The number of available CPUs is: 2
# Retrieve memory configuration information
from psutil import virtual_memory
ram_gb = virtual_memory().total / 1e9
print('Your runtime has {:.1f} gigabytes of available RAM\n'.format(ram_gb))
Your runtime has 13.6 gigabytes of available RAM
# Retrieve GPU configuration information
gpu_info = !nvidia-smi
gpu_info = '\n'.join(gpu_info)
print(gpu_info)
Fri Nov 4 17:08:30 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 69C P0 29W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
# Set the random seed number for reproducible results
RNG_SEED = 888
import random
random.seed(RNG_SEED)
import numpy as np
np.random.seed(RNG_SEED)
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import os
import sys
import math
# import boto3
import zipfile
from datetime import datetime
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
import tensorflow as tf
tf.random.set_seed(RNG_SEED)
from tensorflow import keras
from tensorflow.keras.callbacks import ReduceLROnPlateau
from tensorflow.keras.preprocessing.image import ImageDataGenerator
# Begin the timer for the script processing
START_TIME_SCRIPT = datetime.now()
# Set up the number of CPU cores available for multi-thread processing
N_JOBS = 1
# Set up the flag to stop sending progress emails (setting to True will send status emails!)
NOTIFY_STATUS = False
# Set the percentage sizes for splitting the dataset
TEST_SET_RATIO = 0.2
VAL_SET_RATIO = 0.2
# Set the number of folds for cross validation
N_FOLDS = 5
N_ITERATIONS = 1
# Set various default modeling parameters
DEFAULT_LOSS = 'categorical_crossentropy'
DEFAULT_METRICS = ['accuracy']
DEFAULT_OPTIMIZER = tf.keras.optimizers.Adam(learning_rate=0.0001)
CLASSIFIER_ACTIVATION = 'softmax'
MAX_EPOCHS = 3
BATCH_SIZE = 16
# CLASS_LABELS = []
# CLASS_NAMES = []
# RAW_IMAGE_SIZE = (250, 250)
TARGET_IMAGE_SIZE = (299, 299)
INPUT_IMAGE_SHAPE = (TARGET_IMAGE_SIZE[0], TARGET_IMAGE_SIZE[1], 3)
# Define the labels to use for graphing the data
TRAIN_METRIC = "accuracy"
VALIDATION_METRIC = "val_accuracy"
TRAIN_LOSS = "loss"
VALIDATION_LOSS = "val_loss"
# Define the directory locations and file names
STAGING_DIR = 'staging/'
TRAIN_DIR = 'staging/ASL_Alphabet_Dataset/asl_alphabet_train'
# VALID_DIR = ''
TEST_DIR = 'staging/ASL_Alphabet_Dataset/asl_alphabet_test'
TRAIN_DATASET = 'archive.zip'
# VALID_DATASET = ''
# TEST_DATASET = ''
# TRAIN_LABELS = ''
# VALID_LABELS = ''
# TEST_LABELS = ''
# OUTPUT_DIR = 'staging/'
# SAMPLE_SUBMISSION_CSV = 'sample_submission.csv'
# FINAL_SUBMISSION_CSV = 'submission.csv'
# Check the number of GPUs accessible through TensorFlow
print('Num GPUs Available:', len(tf.config.list_physical_devices('GPU')))
# Print out the TensorFlow version for confirmation
print('TensorFlow version:', tf.__version__)
Num GPUs Available: 1 TensorFlow version: 2.9.2
# Set up the email notification function
def status_notify(msg_text):
access_key = os.environ.get('SNS_ACCESS_KEY')
secret_key = os.environ.get('SNS_SECRET_KEY')
aws_region = os.environ.get('SNS_AWS_REGION')
topic_arn = os.environ.get('SNS_TOPIC_ARN')
if (access_key is None) or (secret_key is None) or (aws_region is None):
sys.exit("Incomplete notification setup info. Script Processing Aborted!!!")
sns = boto3.client('sns', aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name=aws_region)
response = sns.publish(TopicArn=topic_arn, Message=msg_text)
if response['ResponseMetadata']['HTTPStatusCode'] != 200 :
print('Status notification not OK with HTTP status code:', response['ResponseMetadata']['HTTPStatusCode'])
if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 1 - Prepare Environment completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 2 - Load and Prepare Images has begun on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
# Clean up the old files and download directories before receiving new ones
!rm -rf staging/
if not os.path.exists(TRAIN_DATASET):
!wget https://dainesanalytics.com/datasets/kaggle-debashishsau-sign-language-aplhabet/archive.zip
zip_ref = zipfile.ZipFile(TRAIN_DATASET, 'r')
zip_ref.extractall(STAGING_DIR)
zip_ref.close()
CLASS_LABELS = os.listdir(TRAIN_DIR)
print(CLASS_LABELS)
NUM_CLASSES = len(CLASS_LABELS)
print('Total number of classes detected:', NUM_CLASSES)
['O', 'F', 'Y', 'X', 'V', 'I', 'P', 'space', 'nothing', 'J', 'R', 'A', 'K', 'B', 'M', 'T', 'D', 'Q', 'N', 'E', 'del', 'L', 'H', 'C', 'G', 'U', 'S', 'Z', 'W'] Total number of classes detected: 29
# Brief listing of training image files for each class
for c_label in CLASS_LABELS:
training_class_dir = os.path.join(TRAIN_DIR, c_label)
training_class_files = os.listdir(training_class_dir)
print('Number of training images for', c_label, ':', len(os.listdir(training_class_dir)))
print('Training samples for', c_label, ':', training_class_files[:5],'\n')
Number of training images for O : 8140 Training samples for O : ['O (29).jpg', 'O (3830).jpg', 'O1036.jpg', 'O2505.jpg', 'O1629.jpg'] Number of training images for F : 8031 Training samples for F : ['F (1504).jpg', 'F937.jpg', 'F (3480).jpg', 'F1587.jpg', 'F (2422).jpg'] Number of training images for Y : 8178 Training samples for Y : ['Y (3607).jpg', 'Y2686.jpg', 'y_41_rotate_2.jpeg', 'Y944.jpg', 'Y (2911).jpg'] Number of training images for X : 8093 Training samples for X : ['X414.jpg', 'X454.jpg', 'X (2250).jpg', 'X (1825).jpg', 'X (3878).jpg'] Number of training images for V : 7597 Training samples for V : ['V (1473).jpg', 'V1471.jpg', 'V (1164).jpg', 'V (1160).jpg', 'V1197.jpg'] Number of training images for I : 7953 Training samples for I : ['I (1047).jpg', 'I1098.jpg', 'i_67_rotate_4.jpeg', 'I2583.jpg', 'I (2367).jpg'] Number of training images for P : 7601 Training samples for P : ['P (1331).jpg', 'P2084.jpg', 'P (62).jpg', 'P (2728).jpg', 'P1842.jpg'] Number of training images for space : 7071 Training samples for space : ['space1973.jpg', 'space (3307).jpg', 'space1479.jpg', 'space (1085).jpg', 'space (3057).jpg'] Number of training images for nothing : 3030 Training samples for nothing : ['nothing1763.jpg', 'nothing1086.jpg', 'nothing1391.jpg', 'nothing783.jpg', 'nothing471.jpg'] Number of training images for J : 7503 Training samples for J : ['j_27_rotate_7.jpeg', 'J1931.jpg', 'J (1316).jpg', 'J2703.jpg', 'J1564.jpg'] Number of training images for R : 8021 Training samples for R : ['R (3978).jpg', 'R (613).jpg', 'R (1870).jpg', 'R2176.jpg', 'R2411.jpg'] Number of training images for A : 8458 Training samples for A : ['A96.jpg', 'A (3334).jpg', 'A30.jpg', 'A (3943).jpg', 'A (1809).jpg'] Number of training images for K : 7876 Training samples for K : ['K (72).jpg', 'K2571.jpg', 'k_10_rotate_3.jpeg', 'K1516.jpg', 'K2386.jpg'] Number of training images for B : 8309 Training samples for B : ['B (1499).jpg', 'B125.jpg', 'B (429).jpg', 'B (2945).jpg', 'B (3436).jpg'] Number of training images for M : 7900 Training samples for M : ['M (882).jpg', 'M2458.jpg', 'm_59_rotate_8.jpeg', 'M (3435).jpg', 'M254.jpg'] Number of training images for T : 8054 Training samples for T : ['T346.jpg', 'T2210.jpg', 'T494.jpg', 'T (874).jpg', 'T1384.jpg'] Number of training images for D : 7629 Training samples for D : ['D603.jpg', 'D (310).jpg', 'D (359).jpg', 'D (2548).jpg', 'D (784).jpg'] Number of training images for Q : 7954 Training samples for Q : ['Q1281.jpg', 'q_61_rotate_3.jpeg', 'q_38_rotate_4.jpeg', 'Q (3941).jpg', 'Q1985.jpg'] Number of training images for N : 7932 Training samples for N : ['N2249.jpg', 'n_44_rotate_3.jpeg', 'N0012_test.jpg', 'N (919).jpg', 'N (149).jpg'] Number of training images for E : 7744 Training samples for E : ['E828.jpg', 'E749.jpg', 'E (756).jpg', 'E (1054).jpg', 'E0022_test.jpg'] Number of training images for del : 6836 Training samples for del : ['del880.jpg', 'del796.jpg', 'del (602).jpg', 'del (1448).jpg', 'del (544).jpg'] Number of training images for L : 7939 Training samples for L : ['L2890.jpg', 'L (2095).jpg', 'L (3164).jpg', 'L (3614).jpg', 'L1463.jpg'] Number of training images for H : 7906 Training samples for H : ['H (967).jpg', 'H2558.jpg', 'h_31_rotate_3.jpeg', 'h_13_rotate_2.jpeg', 'H1609.jpg'] Number of training images for C : 8146 Training samples for C : ['C (1707).jpg', 'C1004.jpg', 'c_60_rotate_10.jpeg', 'c_15_rotate_6.jpeg', 'C (3150).jpg'] Number of training images for G : 7844 Training samples for G : ['G (2586).jpg', 'G (2349).jpg', 'G (798).jpg', 'G2503.jpg', 'G (3509).jpg'] Number of training images for U : 8023 Training samples for U : ['U (3741).jpg', 'U (1709).jpg', 'U269.jpg', 'U (1264).jpg', 'U (355).jpg'] Number of training images for S : 8109 Training samples for S : ['S2762.jpg', 'S563.jpg', 'S2659.jpg', 'S (2816).jpg', 'S1002.jpg'] Number of training images for Z : 7410 Training samples for Z : ['Z (694).jpg', 'Z753.jpg', 'Z (828).jpg', 'Z1739.jpg', 'Z655.jpg'] Number of training images for W : 7787 Training samples for W : ['W (960).jpg', 'W (2147).jpg', 'W1504.jpg', 'W2400.jpg', 'W1614.jpg']
# Plot some training images from the dataset
nrows = len(CLASS_LABELS)
ncols = 4
training_examples = []
example_labels = []
fig = plt.gcf()
fig.set_size_inches(ncols * 4, nrows * 3)
for c_label in CLASS_LABELS:
training_class_dir = os.path.join(TRAIN_DIR, c_label)
training_class_files = os.listdir(training_class_dir)
for j in range(ncols):
training_examples.append(training_class_dir + '/' + training_class_files[j])
example_labels.append(c_label)
# print(training_examples)
# print(example_labels)
for i, img_path in enumerate(training_examples):
# Set up subplot; subplot indices start at 1
sp = plt.subplot(nrows, ncols, i+1)
sp.text(0, 0, example_labels[i])
# sp.axis('Off')
img = mpimg.imread(img_path)
plt.imshow(img)
plt.show()
datagen_kwargs = dict(rescale=1./255, validation_split=VAL_SET_RATIO)
training_datagen = ImageDataGenerator(**datagen_kwargs)
validation_datagen = ImageDataGenerator(**datagen_kwargs)
dataflow_kwargs = dict(class_mode="categorical")
do_data_augmentation = True
if do_data_augmentation:
training_datagen = ImageDataGenerator(rotation_range=45,
horizontal_flip=True,
vertical_flip=True,
**datagen_kwargs)
print('Loading and pre-processing the training images...')
training_generator = training_datagen.flow_from_directory(directory=TRAIN_DIR,
target_size=TARGET_IMAGE_SIZE,
batch_size=BATCH_SIZE,
shuffle=True,
seed=RNG_SEED,
subset="training",
**dataflow_kwargs)
print('Number of training image batches per epoch of modeling:', len(training_generator))
print('Loading and pre-processing the validation images...')
validation_generator = validation_datagen.flow_from_directory(directory=TRAIN_DIR,
target_size=TARGET_IMAGE_SIZE,
batch_size=BATCH_SIZE,
shuffle=False,
subset="validation",
**dataflow_kwargs)
print('Number of validation image batches per epoch of modeling:', len(validation_generator))
Loading and pre-processing the training images... Found 178472 images belonging to 29 classes. Number of training image batches per epoch of modeling: 11155 Loading and pre-processing the validation images... Found 44602 images belonging to 29 classes. Number of validation image batches per epoch of modeling: 2788
if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 2 - Load and Prepare Images completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 3 - Define and Train Models has begun on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
# Define the function for plotting training results for comparison
def plot_metrics(history):
fig, axs = plt.subplots(1, 2, figsize=(24, 15))
metrics = [TRAIN_LOSS, TRAIN_METRIC]
for n, metric in enumerate(metrics):
name = metric.replace("_"," ").capitalize()
plt.subplot(2,2,n+1)
plt.plot(history.epoch, history.history[metric], color='blue', label='Train')
plt.plot(history.epoch, history.history['val_'+metric], color='red', linestyle="--", label='Val')
plt.xlabel('Epoch')
plt.ylabel(name)
if metric == TRAIN_LOSS:
plt.ylim([0, plt.ylim()[1]])
else:
plt.ylim([0, 1])
plt.legend()
# Define the baseline model for benchmarking
def create_nn_model(input_param=INPUT_IMAGE_SHAPE, output_param=NUM_CLASSES, dense_nodes=2048,
classifier_activation=CLASSIFIER_ACTIVATION, loss_param=DEFAULT_LOSS,
opt_param=DEFAULT_OPTIMIZER, metrics_param=DEFAULT_METRICS):
base_model = keras.applications.xception.Xception(include_top=False, weights='imagenet', input_shape=input_param)
nn_model = keras.models.Sequential()
nn_model.add(base_model)
nn_model.add(keras.layers.Flatten())
nn_model.add(keras.layers.Dense(dense_nodes, activation='relu')),
nn_model.add(keras.layers.Dense(output_param, activation=classifier_activation))
nn_model.compile(loss=loss_param, optimizer=opt_param, metrics=metrics_param)
return nn_model
# Initialize the neural network model and get the training results for plotting graph
start_time_module = datetime.now()
tf.keras.utils.set_random_seed(RNG_SEED)
baseline_model = create_nn_model()
baseline_model_history = baseline_model.fit(training_generator,
epochs=MAX_EPOCHS,
validation_data=validation_generator,
verbose=1)
print('Total time for model fitting:', (datetime.now() - start_time_module))
Epoch 1/3 11155/11155 [==============================] - 7579s 678ms/step - loss: 0.2965 - accuracy: 0.9152 - val_loss: 0.4181 - val_accuracy: 0.9009 Epoch 2/3 11155/11155 [==============================] - 7564s 678ms/step - loss: 0.0417 - accuracy: 0.9894 - val_loss: 0.5608 - val_accuracy: 0.8894 Epoch 3/3 11155/11155 [==============================] - 7541s 676ms/step - loss: 0.0306 - accuracy: 0.9923 - val_loss: 0.5025 - val_accuracy: 0.8994 Total time for model fitting: 6:18:16.928629
baseline_model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
xception (Functional) (None, 10, 10, 2048) 20861480
flatten (Flatten) (None, 204800) 0
dense (Dense) (None, 2048) 419432448
dense_1 (Dense) (None, 29) 59421
=================================================================
Total params: 440,353,349
Trainable params: 440,298,821
Non-trainable params: 54,528
_________________________________________________________________
plot_metrics(baseline_model_history)
# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 3 - Define and Train Models completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 4 - Tune and Optimize Models has begun on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
# Not applicable for this iteration of modeling
# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 4 - Tune and Optimize Models completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 5 - Finalize Model and Make Predictions has begun on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
# Not applicable for this iteration of modeling
# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 5 - Finalize Model and Make Predictions completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))
print ('Total time for the script:',(datetime.now() - START_TIME_SCRIPT))
Total time for the script: 6:19:53.276701